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The Effect of Participation in Undergraduate Research on Critical Thinking and 

Reflective Judgment 

Karen W. Bauer 
The University of Delaware 



Abstract 

The study examined differences in freshman year personality type and the effects 
of participation in undergraduate research on critical thinking and reflective judgment 
scores. While correlations between personality and the thinking measures were low and 
of little practical significance, a repeated measures analysis revealed a two-way 
interaction between research participation and major for change in critical thinking 
score. A second repeated measures analysis with simple contrasts revealed a two-way 
interaction between research participation and gender for change in reflective judgment 
score. Findings thus indicate that participation in undergraduate research effects 
critical thinking and reflective judgment for some students. Implications and limitations 
are discussed. 
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In recent years, there have been numerous calls for focused attention and reform 
in the research university to bolster student achievement, and in particular, critical 
thinking skills (National Commission on Excellence in Education, 1983; American 
Association of Colleges, 1985). The authors of Reinventing Undergraduate Education 
(Boyer Commission, 1998), urged research universities to “restructure the pedagogical 
and integrative aspects of the research university experience” (p. 1), in part, by making 
research-based learning the standard. These notions of inquiry-based learning enable 
faculty to guide students by mentoring and acknowledge that learning is a reciprocal 
process whereby both students and faculty learn from each other. 

Reinventing Undergraduate Education, was designed to be not another document 
deploring the state of undergraduate education but rather a guide, a “blueprint,” for the 
way forward into integrating research in education; the way toward “reinvention,” the 
report says, is to “make research-based learning the standard” for undergraduate 
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education at research universities (Boyer, 1998, pp. 15-16). Historically, research and 
education have been closely integrated for undergraduates who collaborate with faculty 
to conduct research, and this practice underlies the “model” that the Boyer Commission 
proposes, with “scholar-teachers” who “would treat the sites of their research as seminar 
rooms in which not only graduate students but undergraduates observe and participate in 
the process of both discovery and communication of knowledge” (Boyer, 1998, p. 18). 

Among college outcomes, achievement has been one of the most frequently 
researched topics in higher education (Astin, 1977, 1993). Similarly, critical thinking 
skills are regarded as one of the major outcomes of college education (Pascarella, 1989; 
Pascarella & Terenzini, 1991). Pascarella, et al. (1989) found that college experiences 
were modestly associated with higher critical thinking skills, while the composite college 
experience scale (e.g., type of course work, non-classroom interaction with faculty, study 
time, and extra-curricular activities) showed significant partial correlation with overall 
critical thinking skills. Facione, Sanchez, Facione, and Gainen (1995) also argue that 
critical thinking dispositions are an important outcome of higher education and The 
American Association of Colleges and Universities commission report discusses the 
importance of reasoning about real-world issues as a central goal of higher education in 
the publication, The Challenge of Connecting Learning'. 

“In the final analysis, the real challenge of college students and faculty members 
alike is empowering individuals to know that the world is far more complex than 
it first appears and that they must make interpretive arguments and decision- 
judgments that entail real consequences for which they must take responsibility 
and from which they may not flee by disclaiming expertise." (AAC, 1990) 
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Cognitive Development 

Based on Perry’s (1970) theory of ethical and intellectual development, King and 
Kitchener (1994) Wood (1997a), Rnefelkamp (1974) and Widick (1975), Baxter- 
Magolda (1985), have applied cognitive development theory to college students. In a 
cross sectional data analysis of The RneWi, a measure of intellectual development, 
Heidke and Omahan (1981) found seniors to score higher than freshmen. With a semi- 
structured interview, the Measure of Epistemological Reflection (MER), Baxter-Magolda 
(1987) examined six domains of thinking related to learning. Each domain is measured 
by a series of questions that follow a general question intended to focus the respondents’ 
thinking on the domain. While Baxter-Magolda (1988) found that both men and women 
“demonstrate similar cognitive structures regardless of learning style” (p. 533), she also 
reported (1987; Baxter-Magolda and Porterfield (1985) significant increases in MER 
score from freshmen to senior years. 

In a series of studies, Kitchener and King (1981) and colleagues King and Parker 
(1978, in Welfel, 1982), Lawson (1980), Strange and King (1982) and King, Kitchener, 
and Wood (1994) have examined the seven-stage theory of reflective judgment. The 
semi-structured interview, the Reflective Judgment Interview (RJI), and the paper-pencil 
instrument, Reasoning About Current Issues (RCI; Wood, 1997a) have established a 
positive correlation between education level and reflective judgment score. Consistent 
with findings of King, Kitchener, Davison, Parker, and Wood (1983), Welfel and 
Davison (1986) found a significant increase in reflective judgment stage for a sample of 
25 students from the freshman to senior years (about one-half of one stage). Further, 
Welfel and Davison reported no significant difference in RJI score by gender, or time by 
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major interaction. Related to the developmental nature reflective judgment, Welfel 
(1982) notes two important points: 1). higher stages of reflective judgment cannot be 
explained by maturation alone; and 2). measures of critical thinking and verbal ability 
are associated but do not account for differences between groups in reflective judgment. 

Although research has revealed a significant relationship between education level and 
reflective judgment score, it is not known which factors within the educational institution 
influence this relationship. For example. Kitchener and King (1981) suggested that 
academic major may be related to reflective judgment. Similarly^ co-curricular activities 
such as major-related internships and undergraduate research may be positively affecting 
one’s reflective judgment score. Thus, the value in further investigation of reflective 
judgment is warranted. 

Use Of Personality Type to Help Assess Undergraduate Success 

While personality type has been used to predict occupational performance (Mount 
& Barrick, 1989; Hough, 1992), the use of personality type to assist in college student 
success has recently received more attention as an additional factor to help assess 
collegiate success. Entwistle’s (1972) review of studies involving Cattell’s 16 
Personality Factors and Eysenck’s Personality Inventory concluded that college success 
is associated with introversion, and Entwistle & Entwistle (1970) partly attribute 
introverts’ higher academic achievement to their good study habits. Additionally, 

Digman & Takemoto-Chock (1981) found that conscientious students are well-organized, 
purposeful, and persistent, and that these characteristics are highly related to academic 
achievement (e.g., GPA). In a 45-year longitudinal study of 63 men, Soldz and Vaillant 
(1999) found that openness to experiences was strongly related to creativity and that 
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conscientiousness in college was the best predictor of what happened to men in the 
future. Tross, Harper, Osher, and Kneidinger (2000) found that conscientiousness was 
related to retention and contributed significantly to the prediction of college GPA. 
Participation in Undergraduate Research 

Student involvement in research is an effective way to heighten inquiry-based 
learning. This involvement in research typically manifests itself in the colleagial and 
synergystic relationships that develop between undergraduate student, graduate students, 
and faculty members. Recognizing this benefit, the National Science Foundation Report 
(1989) stated “it is clear that the academic community regards the involvement of 
undergraduate student majors in meaningful research . . . with faculty members as one of 
the most powerful instruction tools” (p. 6). According to Hakim (1998), there are four 
features inherent in undergraduate research (UR) experiences: mentorship, originality, 
acceptability, and dissemination. These features assume that students will be involved in 
meaningful contributions and will be using methods consistent with current practices 
within the discipline. UR experiences also include interactions between the faculty and 
student researcher focus on student learning, and experiences that will culminate in a 
tangible product that is critiqued by others in the discipline. Thus, undergraduate 
research students get the opportunity to develop both practical and complex thinking and 
problem-solving, technical competence, and leadership skills. These are important tasks 
that strongly contribute to the undergraduate experience and successful attainment of the 
baccalaureate degree. 
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Research Questions for Current Study 

In an attempt to better understand factors that contribute to high quality 
undergraduate education, this researcher sought to understand the effect of participation 
in UR on critical thinking and reflective judgment for undergraduates at one Mid- Atlantic 
doctoral-extensive university. The research questions are: 

1) What role does participation in undergraduate research play in the change in critical 
thinking scores (as measured through the Watson Glaser Critical Thinking Appraisal and 
the Reasoning About Current Issues test)? 

2) . Are there differences in these outcomes by major or gender? and 

3) . Does personality (as measured by NEO-FFI in the freshman year) differ for research 
versus non-research students? 

Participants 

Participants were 266 undergraduate students at a Research-Extensive state 
university who were part of a four-year longitudinal study funded by the National 
Science Foundation to the benefits of participation in undergraduate research. Students 
were enrolled in 47 majors, which were grouped into three categories: 
sciences/math/nursing; chemistry and engineering; and psychology and other majors. 
Among them were 155 females (58%) and lll males (42%), 202 White (76%) and 64 
non- Whites (24%). 

Instruments 

Information about these students were collected from the university’s student 
records database and by responses obtained from the following three published measures: 
the Watson-Glaser Critical Thinking Appraisal (WGCTA; Watson & Glaser, 1994), the 
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Reasoning About Current Issues Test (Wood, 1987), and the NEO-Five Factor Inventory 
(NEO-FFI; Costa, Jr.& McCrae, 1991). 

Watson Glaser Critical Thinking Appraisal (WGCTAT The WGCTA is a 
composite measure that examines attitudes of inquiry, knowledge of the nature of valid 
inferences, abstractions, and generalizations, and skills using attitudes and knowledge 
(Watson & Glaser, 1994). The WGCTA Form S consists of 40 items measuring five 
subtests of critical thinking: 1) inference-discriminating the truth or falsity of inferences 
drawn from given data; 2) recognition of assumption— recognizing unstated assumptions 
or presuppositions in given statements or assertions; 3) deduction-determining whether 
certain conclusions necessarily follow from information in given statements or premises; 
4) interpretation— weighing evidence and deciding if generalizations or conclusions based 
on the given data are warranted; and 5) evaluation of arguments— distinguishing between 
arguments that are strong and relevant and those that are weak or irrelevant to a particular 
question at are strong and relevant and those that are weak or irrelevant to a particular 
question at issue (Watson & Glaser, 1994, pp. 9-10). The WGCTA was completed in the 
freshman and senior year, and has a reliability of 0.77 for this sample. 

Reasoning About Current Issues (RCI). The RCI is a paper-pencil measure 
designed to measure the epistemologic construct, reflective judgment. Reflective 
judgment research seeks to document progressive sophistication in the way that 
individuals reason about a particular type of problem termed an “ill-structured problem.” 
The RCI is composed of two parts, each scored separately. These two parts are: 

IV Student Essay Discrimination Measure . The first half of the RCI asks 
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students to play the role of an instructor who has assigned students “minute essays” about 
a real world topic. Students are presented with pairs of such essays and are asked to 
judge which student is reasoning about the issue in a more complex or sophisticated 
fashion. This part of the instrument is designed to assess whether students can identify 
which of a pair of statements is a more sophisticated justification for a position. 

21. RCI Dilemmas: Student Endorsement of Justifications . The second half of 
the RCI presents students with five real-world issues as well as statements written to 
reflect arguments at varying levels of sophistication according to the Reflective Judgment 
Model. Students are asked to indicate how similar each of these statements are to their 
own thinking about the issue, and are then asked to rank the three statements which are 
most similar to their own reasoning about the issue. 

NEO-Five Factor Inventory (NEO-FFD. This instrument measures the most basic 
dimensions underlying human traits (Costa & McCrae, 1991). There are five subtests in 
this inventory, each composed of twelve items. The five subtests are: 1) NEO- 
Neuroticism which measures an individual’s level of adjustment and emotional stability 
(coefficient alpha= 0.86); 2) NEO-Extroversion measures level of sociability and 
consequent behaviors that occur as a result of interactions with others (coefficient alpha= 

0.81); 3) NEO-Openness to Experiences measures imagination, aesthetic sensitivity, 
attentiveness to inner feelings, preference for variety, intellectual curiosity, and 
independence of judgment (coefficient alpha= 0.73); 4) NEO-Agreeableness measures 
level of sympathy and altruism toward others, eagerness to help (coefficient alpha= 

0.77); and 5) NEO-Conscientiousness measures ability to manage impulses and desires 
and the process of planning, organizing, and carrying out tasks (coefficient alpha= 0.83). 



O 

ERIC 



10 



Effect of Research on Critical Thinking and Reflective Judgment 9 

Procedure 

After receiving approval from the University’s Human Subjects Committee, 
researchers sent a letter in March, 1997 to second semester freshmen students majoring in 
science, math, and psychology requesting their participation in this four-year longitudinal 
study of their academic experiences. Two hundred sixty-six students agreed to 
participate and met with the researchers to complete several questionnaires including the 
WGCTA and NEO-FFI. Students were also asked to complete the Reasoning About 
Current Issues (RCI) beginning in Spring 1998. Each survey took approximately 15 to 
30 minutes to complete. A signed consent form also enabled the researchers to obtain 
demographic data from the university’s student record system (i.e., high school GPA, 
sex, ethnic classification, SAT, and cumulative GPA). During each survey completion, 
students were offered a small monetary incentive ($5 in Spring 1997, $10 in 1998; $15 in 
1999; and $20 in Spring 2000). 

Characteristics of the Undergraduate Research Program and Definition of Participation in 
Undergraduate Research 

Undergraduate Research Program opportunities, which are open to all interested 
undergraduates, include research collaborations that occur during the academic year as 
well as summer research, senior thesis, and oral presentations of research-in-progress to 
other students and faculty. Over 66 percent of the University’s approximately 900- 
member faculty regularly participate inUR, including over 90 percent of the faculty in the 
sciences and engineering. From 600 to 700 students are estimated to be engaging in UR 
at any one time. Students can begin research at any point in their undergraduate careers; 
however, normal starting time is mid-sophomore year. Student involvement generally 
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lasts for one and one-half to two or more years and usually includes one or more summer 
or winter terms in addition to the academic semesters. 

Each spring, students reported the number of hours per week they had been 
involved in a variety of campus activities, including undergraduate research. Student 
self-reports were verified with faculty and then summed to reveal the total number of 
hours over the four baccalaureate years each student had participated in UR. The total 
number of hours for participation in research ranged from 0 to 3,342; the mean number of 
hours for those who participated in research was 626 hours, SD = 596.5. Based on a 
scatterplot of research hours, students were then grouped into three categories: 1). no 
research; 2). total research less than 700 hours; and 3). total research 700 or more hours. 

Results 

Table 1 shows descriptive statistics and Table 2 highlights the correlation matrix 
for the primary measures in this study. Shown in Table 2 and as expected, SAT is 
strongly correlated with first year cumulative grade point average. Also, WGCTA score 
is significantly correlated with SAT, but RCI score is not highly correlated with SAT. 
Although the correlation coefficients are significant, the values between .21 and .29 do 
not reveal strong relationships. An independent samples T-test revealed significant 
differences between research and non-research students for NEO-N (t=2.81, p =.005) and 
NEO-E (t=l .98, p = .048). These findings indicate that students who later participated in 
UR scored higher on the neuroticism and lower on the extroversion scales as freshmen 
than peers who did not participate in UR. 
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Table 1 

Descriptive Statistics for WGCTA, RCI, and NEO FFI 





M Year 1 


M Year 2 


M Year 3 


M Year 4 


WGCTA 


29.47 ( 5 . 33 ) 






30.12 ( 5 . 22 ) 


RCI 




5.26 (. 62 ) 


5.31 (.66) 


5.28 (. 67 ) 


NEO-N 


21.90 (8.22) 








NEO-E 


30.51 ( 6 . 54 ) 








NEO - 0 


30.37 ( 6 . 13 ) 








NEO -A 


32.33 (6.12) 








NEO-C 


32.72 ( 6 . 72 ) 









Changes in Critical Thinking 

Repeated measures analyses were performed to examine the effect of research 
participation on changes in critical thinking and reflective judgment. Table 3 illustrates 
the effects of research, major and gender on the outcome variable, WGCTA. Because of 
the high correlation between SAT and WGCTA, SAT was included as a covariate in the 
analysis for WGCTA. As shown in Table 3, major (F=4.129, p <.01) and the interaction 
and major and research (F=3.094, p <.02) had an effect on the WGCTA change score. 
Changes in WGCTA were not effected by gender or interactions of gender with major or 
gender with research. Figures 1 to 3 illustrate the interaction of research and major on 
WGCTA. As shown, WGCTA change is greatest for students who participated in 800 or 
more hours of research, but this change differs by major. As shown in the figures, 
freshman to senior year WGCTA scores increase for science/math/nursing and 
chemistry/engineering majors, but decreases for psychology/other majors. 
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Table 3 

Repeated Measures ANOVA for Watson-Glaser Critical Thinking Appraisal 



Measure 


Sum Sa 


df 


MS 


F 


WGCTA 


5.960 


1 


5.960 


.362 


WG x Research 


5.270 


2 


2.635 


.160 


WG x SAT 


2.970 


1 


2.970 


.180 


WG x Gender 


28.806 


1 


28.806 


1.748 


WG x Research x Gender 


75.130 


2 


37.565 


2.280 


WG x Major 


136.081 


2 


68.041 


4.129* 


WG x Research x Major 


203.929 


4 


50.982 


3.094 * 


Error 


3163.578 


192 


16.477 





* p < .02 



Figures 1-3 

Estimated Marginal Means for WGCTA by Research Participation and Major 



Estimated Marginal Means of Watson-Glaser 
At major= science, nurs, & math 



Estimated Marginal Means of Watson-Glasei 
At major = chem & engin 




WG 



WG 



Estimated Marginal Means of Watson-Glaser 
At major= psych & othr 



2 

2 



2 

* 

1 




WG 
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263 
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0.09 

0.16 

263 


-0.01 

0.92 

263 


NEO-E 


















0.01 

0.84 

263 


0.32 

0.00 

263 


0.23 

0.00 

263 


NEO-N 
















-0.39 

0.00 

263 


-0.05 

0.40 

263 


-0.17 

0.01 

263 


-0.25 

0.00 

263 


Yr4 RCI 














-0.13 

0.07 

205 


0.01 

0.86 

205 


0.17 

0.02 

205 


0.19 

0.01 

205 


-0.04 

0.53 

205 


Yr3 RCI 
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-0.07 

0.35 

200 


0.06 

0.38 

200 


0.15 

0.04 

200 


0.25 

0.00 

200 


-0.09 

0.18 

200 


Yr2 RCI 










0.52 

0.00 

203 


0.46 

0.00 

195 


-0.08 

0.21 

227 


0.06 

0.38 

227 


0.29 

0.00 

227 


0.22 

0.00 

227 


-0.01 

0.89 

227 


Yr 2 WG 








0.25 

0.00 

196 


0.35 

0.00 

181 


0.27 

0.00 

202 


-0.01 

0.86 

209 


-0.16 

0.02 

209 


0.07 

0.33 

209 


0.11 

0.12 

209 


-0.30 

0.00 

209 


TO 


Yrl WG 






0.67 

0.00 

210 


0.22 

0.00 

228 


0.34 

0.00 

201 


0.23 

0.00 

206 


-0.08 

0.17 

263 


-0.17 

0.01 

263 


0.15 

0.02 

263 


0.09 

0.16 

263 


-0.23 

0.00 

263 


total SAT 
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0.42 

0.00 

260 


0.25 

0.00 

264 


0.22 

0.00 

212 


0.09 

0.18 

230 


0.09 

0.18 

203 


0.06 

0.39 

208 


0.00 

0.94 

263 


-0.17 

0.00 

263 


-0.02 

0.80 

263 


0.06 

0.34 

263 


0.22 

0.00 

263 




Pearson Co it. 
Sig. (2-tailed) 
N 


Pearson Corr. 
Sig. (2-tailed) 
N 


Pearson Corr. 
Sig. (2-tailed) 
N 


Pearson Corr. 
Sig. (2-tailed) 
N 


Pearson Corr. 
Sig. (2-tailed) 
N 


Pearson Corr. 
Sig. (2-tailed) 
N 


Pearson Corr. 
Sig. (2-tailed) 
N 


Pearson Corr. 
Sig. (2-tailed) 
N 


Pearson Corr. 
Sig. (2-tailed) 
N 


Pearson Corr. 
Sig. (2-tailed) 
N 


Pearson Corr. 
Sig. (2-tailed) 
N 
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Change in Reflective Judgment Score 

Table 4 shows the effects of research, major and gender on the outcome variable 
for reflective judgment, Reasoning About Current Issues. A separate model including 
NEO-N and NEO-E as covariates was run; no substantial differences were shown, thus 
the analysis presented herein does not include NEO scores as covariates. As shown in 
Table 4 using a simple comparisons contrast, participation in research had a main effect 
on RCI change when comparing change from the sophomore to senior year (F=3.818, 
p=.05). Similarly, the interaction of gender and research had an effect on RCI change 
from the sophomore to senior year (F=3.063, p=. 04). Figures 4 and 5 illustrate the 
interaction between RCI, research participation and gender. 



Table 4 

Repeated Measures ANOVA for Reasoning About Current Issues Test 



Measure 






SS 


df 


MS 


F 


RCI 


level 2 vs 1 


.743 


1 


.743 


1.945 




level 3 vs 1 


1.626 


1 


1.626 


3.818 


RCI x Rsrc 




2 vs 1 


.085 


2 


.042 


.111 






3 vs 1 


.604 


2 


.302 


.709 


RCI x Gendr 




2 vs 1 


.196 


1 


.196 


.513 






3 vs 1 


.850 


1 


.850 


1.997 


RCI x Rsrc x 


Gndr 


2 vs 1 


1.673 


2 


.837 


2.190 






3 vs 1 


2.608 


2 


1.304 


3.063 


RCI x Major 




2 vs 1 


.434 


2 


.217 


.568 






3 vs 1 


2.272 


2 


1.136 


2.668 


RCI x Rsrc x 


Maj 


2 vs 1 


.553 


4 


.138 


.362 






3 vs 1 


1.220 


4 


.305 


.716 


Error 




2 vs 1 


64.178 


168 


.382 








3 vs 1 


71.530 


168 


.426 





* p < .05 
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Figures 4 and 5 

Estimated Marginal Means of RCI by Gender and Research 



Estimated Marginal Means of RCI 
At GENDER OF STUDENT = FEMALE 




Estimated Marginal Means of RCI 
At GENDER OF STUDENT = MALE 




RCI 



RCI 




Discussion 

This study examined the effects of participation in undergraduate research on two 
college outcomes, critical thinking and reflective judgment. Based on findings from this 
sample of 266 students, results indicate that involvement in research does affect critical 
thinking score and, to a lesser degree, reflective judgment. Students who participated in 
research scored higher on NEO-Neuroticism and lower on NEO-Extroversion as 
freshmen than peers who did not participate in research. However, correlations between 
NEO scales and the outcomes measures were less than .30 and determined to be of little 
practical significance. Results from repeated measures analyses indicated that the change 
in WGCTA score from freshman to senior year was greatest for science/math/nursing and 
chemistry/engineering majors who participated in 800 or more hours of undergraduate 
research. The interaction of research participation and gender had an effect on RCI 
change, thus indicating that females who participated in 800 or more hours of research 
and males who participated in 1 -799 hours of research showed the greatest increases in 
RCI score over the three year period. 
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Participation in undergraduate research can benefit undergraduate students by 
contributing to the development of complex thinking, evaluation, and communication 
abilities, time management skills, and enhancing interpersonal independence. Increases 
in these skills will likely increase the students’ chances for success as well as increase 
level of satisfaction with the undergraduate experience. 

Limitations of this study included the relatively small size of the sample 
and selective inclusion of only certain majors. If the sample were larger, it is possible 
that additional relationships may emerge. Like King, et al, (1983) we found some of our 
students to regress in RCI score. Additional study of the pattern of epistemological 
development is warranted. Due to the selective sample, findings from the study are 
limited in generalizability to science and engineering majors.. Findings from this study 
are also unique to the set of academic and social experiences, including undergraduate 
research, that are available at this institution. Additional studies with a larger, random 
sample may be helpful in examining the effects of academic experiences on critical 
thinking and reasoning outcomes. 
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